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16.  Abstract 

As  a  follow-up  to  the  completion  of  the  Concurrent  Validation  of  the  Air  Traffic  Selection  and  Training 
(AT-SAT)  test  for  Tower  Controller  Hiring  (CoVATCH)  project,  we  investigated  the  utility  and  fairness  of 
using  AT-SAT  scores  to  place  Air  Traffic  Control  Specialist  (ATCS)  applicants  into  terminal  or  en  route 
facilities.  Generally  speaking,  controller  pay  is  higher  at  en  route  facilities  than  at  most  terminal  facilities; 
placement  by  option  thus  affects  the  terms  and  conditions  of  employment  and  therefore  is  a  selection  decision 
within  the  scope  of  the  Uniform  Guidelines  on  Employee  Selection  Procedures  (Equal  Employment 
Opportunity  Commission,  1978).  While  results  of  statistical  analyses  indicated  that  AT-SAT  could  be 
considered  a  valid  tool  for  use  in  placement,  based  on  technical  considerations  only,  it  was  concluded  that  it 
should  not  be  used  in  that  way  due  to  lack  of  utility  and  potential  for  adverse  impact.  The  analyses  indicated 
that  if  AT-SAT  were  used  for  placement,  while  the  field  training  success  rate  (the  proportion  of 
developmentals  achieving  Certified  Professional  Controller  status  at  the  first  field  facility)  would  increase  in 
en  route  facilities,  the  success  rate  would  decrease  in  terminal  facilities.  Since  more  positions  are  available  at 
terminal  facilities,  the  overall  success  rate  across  both  options  would  not  change  substantially  and  might 
actually  decrease  slightly.  Furthermore,  using  AT-SAT  to  place  new  ATCSs  into  en  route  or  terminal 
facilities  might  have  adverse  impact  on  blacks,  Hispanics,  and  females.  If  the  FAA  were  to  use  AT-SAT  for 
placement,  the  risk  of  additional  adverse  impact  and  pay  disparities  should  be  evaluated  against  the  marginal 
utility  of  placement  in  terms  of  changes  in  field  training  success  rates.  In  sum,  given  the  findings  of  both 
validation  studies,  the  analyses  conducted  here,  and  the  projected  ratio  of  controllers  that  will  likely  be  hired 
into  each  option,  using  AT-SAT  scores  to  guide  placement  decisions  is  not  recommended  at  this  time. 
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An  Evaluation  of  the  Utility  of  AT-SAT  for  the 
Placement  of  New  Controllers  by  Option 


The  air  traffic  control  specialist  (ATCS,  or  controller)  occupa¬ 
tion  is  considered  to  be  an  intellectually  challenging,  important, 
and  prestigious  career  field  by  the  majority  of  recently  hired  de¬ 
velopmental  controllers  (Cannon  &  Broach,  2011).  The  Federal 
Aviation  Administration  (FAA)  projects  hiring  approximately 
1,300  new  controllers  per  year  over  the  next  five  years  to  replace 
retiring  controllers  (FAA,  2014).  Excluding  rehires  or  others  with 
previous  ATCS  experience,  it  is  required  that  applicants  receive  a 
passing  score  on  an  aptitude  test  to  be  hired  into  the  occupation 
(U.  S.  Office  of  Personnel  Management  [OPM]  ,2013).  Currently, 
the  computer-administered  Air  Traffic  Selection  andTraining  (AT- 
SAT)  test  battery  is  the  aptitude  test  used  by  the  FAA  to  assess 
applicants  under  the  OPM  occupational  qualification  standards. 

The  validity  of  AT-SAT  as  a  predictor  of  ATCS  job  performance 
was  demonstrated  in  two  concurrent,  criterion-related  validation 
studies.  The  first  study  was  reported  in  200 1  by  Ramos,  Heil,  and 
Manning  (2001a,  b).  Approximately  1,000  incumbent  en  route 
controllers  took  the  proposed  test  battery.  Job  performance  data 
were  collected  concurrently  in  two  forms:  Behavioral  Summary 
Scale  (BSS)  ratings  of  job  performance  by  peers  and  supervisors; 
and  the  en  route  Computer-Based  Performance  Measure  (CBPM; 
see  Borman  et  al.,  2001).  These  measures  were  combined  into  a 
composite  score  with  a  60%  contribution  from  the  CBPM  and  a 
40%  contribution  from  the  BSS  ratings.  The  correlation  between 
scores  on  the  test  battery  and  the  composite  job  performance 
measure  was  .51  without  any  corrections  for  range  restriction 
or  criterion  unreliability.  With  correction  for  incidental  range 
restriction,  the  correlation  was  .68  (Waugh,  2001).  The  Ameri¬ 
can  Institutes  for  Research  (AIR®,  2012)  conducted  the  second 
study,  named  the  Concurrent  Validation  of  AT-SAT  for  Tower 
Controller  Hiring  (CoVATCH).  Incumbent  air  traffic  control 
tower  (ATCT)  controllers  (N=  302)  took  the  current  operational 
version  of  the  AT-SAT  test  battery.  As  in  the  original  en  route 
validation  study,  two  classes  of  job  performance  data  were  col¬ 
lected:  BSS  ratings  of  job  performance  by  peers  and  supervisors; 
and  performance  on  the  Tower  Simulation-Based  Performance 
Measure  (TSBPM)  (see  Horgen  et  al.,  2012).  The  correlation 
between  a  regression  weighted  composite  of  AT-SAT  subtest 
scores  and  the  composite  of  the  two  criterion  measures  was  .42 
without  any  statistical  corrections  (AIR®,  2012).  These  two  studies 
independently  demonstrated  that  AT-SAT  is  a  valid  predictor  of 
ATCS  job  performance. 

Placement  of  new  controllers  has  been  a  continuing  concern 
in  Congressional  hearings  and  oversight  with  relatively  recent 
recommendations  to  consider  the  use  of  AT-SAT  for  placement 
purposes  (U.  S.  Department  of  Transportation  Office  of  the 
Inspector  General,  2010).  AT-SAT  scores  are  currently  used  for 
the  selection  of  applicants  into  the  ATCS  occupation,  but  scores 
are  not  currently  being  used  to  place  selected  applicants  into 
different  options  or  career  tracks. 


BACKGROUND 

Before  placement  can  be  discussed,  it  is  useful  to  understand 
the  nature  and  structure  of  the  organization  within  the  FAA 
responsible  for  air  traffic  control  operations  and  facilities.  This 
organization,  called  the  Air  Traffic  Organization,  or  ATO,  can 
be  divided  into  two  major  partitions,  also  referred  to  as  options-. 
Terminal  Services  and  En  Route/Oceanic  Services.  New  hires 
can  be  placed  into  either  the  Terminal  option  or  the  En  Route 
option.  These  options  also  coincide  with  the  primary  training 
tracks  currently  operatingwithin  the  FAA  Academy.  Within  each 
option,  there  are  several  types  of  facilities  to  which  a  new  hire 
could  be  assigned.  In  en  route,  the  vast  majority  of  facilities  are 
air  route  traffic  control  centers  (ARTCCs,  or  en  route  centers), 
but  there  are  also  combination  terminal  radar  approach  control 
(TRACON)/en  route  facilities  call  combined  center  approach 
control  centers  (CERAPs).  On  the  terminal  side,  there  are  airport 
traffic  control  towers  (ATCTs),  stand-alone  TRACONs,  and 
combined  tower/TRACON  (CTT)  facilities. 

At  en  route  centers,  controllers  handle  high  altitude  air  traffic 
between  airports,  work  that  is  generally  considered  very  complex 
and  demanding.  At  TRACONs,  controllers  direct  traffic  within 
about  50  miles  of  an  airport,  usually  during  initial  climb  and 
final  descent  of  the  aircraft.  This  work  can  also  be  considered 
very  demanding.  The  work  at  ATCTs  involves  directing  air  traf¬ 
fic  on  the  runways  and  in  the  immediate  vicinity  of  the  airport, 
as  well  as  issuing  takeoff  and  landing  clearances.  This  type  of 
air  traffic  control  (ATC)  is  generally  considered  somewhat  less 
complex  and  less  demanding  than  radar  ATC,  but  that  can  vary 
greatly  by  location. 

The  FAA  also  classifies  all  facilities  by  level,  from  low  to  high, 
considering,  among  other  things,  the  amount  and  complexity  of 
traffic  controlled  by  the  facility.  Thus,  a  newly  hired  controller 
could  potentially  be  placed  into  a  mid-  to  high-level  en  route 
center,  a  mid-  to  high-level  TRACON,  or  at  any  CTT/ tower- 
only  facility,  from  low  to  high  level,  depending  on  agency  needs. 
With  this  amount  of  variety  in  the  difficulty  and  complexity  of 
the  work,  it  is  becoming  more  and  more  important  for  the  FAA 
to  address  the  issue  of  placement. 

Historically,  the  failure  rate  in  on-the-job  training  for  new 
controllers  has  been  higher  in  en  route  facilities  (Manning, 
1998).  This  difference  in  difficulty  and  complexity  is  evidenced 
in  how  the  positions  in  different  facility  types  are  classified  for 
pay  purposes.  Controller  positions  at  en  route  centers  generally 
have  the  highest  pay  grades  in  the  occupation.  Controller  posi¬ 
tions  in  towers  generally  have  lower  pay  grades  than  en  route 
positions.  Thus,  there  are  both  organizational  (success  and  failure 
rates  in  facility  on-the-job  training)  and  individual  economic 
consequences  attached  to  placement  decisions.  Moreover,  be¬ 
cause  placement  affects  the  terms  and  conditions  of  employment 
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(especially  starting  pay) ,  it  is  an  employment  decision,  as  defined 
by  the  Uniform  Guidelines  on  Employee  Selection  Procedures  (29 
C.F.R.  §  1607. 2B)  (EEOC,  1978).  Therefore,  using  AT-SAT 
scores  for  placement,  as  recommended  by  the  Department  of 
Transportation  Inspector  General  (2010),  must  be  validated. 

To  use  a  test  score  for  placement  purposes,  the  relevant  pro¬ 
fessional  standards  and  principles  require  “evidence  that  scores 
are  linked  to  different  levels  or  likelihoods  of  success  among 
jobs”  (American  Educational  Research  Association,  American 
Psychological  Association,  &National  Council  onMeasurement 
in  Education,  1999,  p.  160).  Relevant  evidence  might  include 
a  pattern  of  differential  relationships  between  predictors  and 
criteria  by  job  type  (Society  for  Industrial  and  Organizational 
Psychology  [SIOP],  2003).  For  example,  the  correlations  be¬ 
tween  AT-SAT  sub-tests  and  job  performance  might  vary  as  a 
function  of  option  (terminal  versus  en  route)  or  level.  Other 
relevant  evidence  might  include  differences  in  expected  success 
and  failure  by  option  as  a  function  of  test  scores.  An  explicit 
purpose  of  the  CoVATCH  project  was  to  collect  empirical  data 
to  support  an  evaluation  of  AT-SAT  as  a  tool  for  placement  by 
option  (terminal  versus  en  route). 

In  their  analysis  of  the  CoVATCEl  data,  AIR®  (20 1 2)  was  not 
able  to  provide  any  evidence  suggesting  that  AT-SAT  could  be 
used  for  placement  by  facility  level.  However,  they  did  report 
evidence  for  validity  by  option  in  that  the  regression  equa¬ 
tion  (i.e.,  the  weight  given  to  each  sub  test)  for  tower  was  not 
identical  with  the  equation  for  en  route,  as  reported  by  Ramos 
et  al.  (  2001a,  b).  Unfortunately,  there  were  few  differences  in 
the  recommended  option  placement  when  the  two  equations 
were  used  to  hypothetically  place  individuals  in  their  sample 
of  300  tower  controllers.  This  finding  was  consistent  with  the 
1995  controller  job-task  analysis  that  found  nearly  identical 
worker  requirement  profiles  across  en  route  and  terminal  op¬ 
tions  (Nickels,  Bobko,  Blair,  Sands,  &Tartak,  1995).  However, 
AIR®  did  not  have  access  to  data  from  en  route  controllers  for 
comparison  purposes.  AIR®  concluded  that  AT-SAT  might  be 
used  for  placement  by  option  but  further  analyses  were  needed. 
Thus,  the  purpose  of  the  current  study  was  to  extend  the  AIR® 
analysis  by  using  AT-SAT  validation  data  collected  from  both 
en  route  and  terminal  controllers. 

PLACEMENT  RULES 

AT-SAT  scores  might  be  used  for  placement  in  many  differ¬ 
ent  ways.  For  example,  persons  with  scores  above  some  cut-off 
might  be  assigned  to  the  en  route  option,  while  persons  with 
scores  below  that  cut-off  might  be  assigned  to  terminal.  Or  scores 
might  be  categorized  into  ranges,  with  persons  in  the  lowest  range 


assigned  to  one  option,  persons  in  the  highest  range  assigned  to 
the  other  option,  and  persons  with  scores  in  the  middle  range 
assigned  to  either  option,  depending  on  agency  needs. 

Thus,  the  first  step  in  this  analysis  was  to  decide  how  AT- 
SAT  scores  would  likely  be  used  for  placement.  At  the  time  of 
this  analysis,  FAA  extended  tentative  job  offers  to  applicants 
for  a  specific  air  traffic  control  facility.  The  decision  to  place  a 
particular  applicant  at  a  specific  facility  was  made  by  members 
of  the  Centralized  Selection  Panel  (CSP) .  This  panel,  made  up  of 
selecting  officials  from  air  traffic  facilities,  was  convened  to  select 
and  place  applicants  based  on  agency  need,  the  geographic  and 
option  (i.e.,  en  route  or  terminal)  preferences  indicated  by  an 
applicant,  and  a  review  of  the  application  -  which  include  AT- 
SAT  score  band  information  (i.e.,  Qualified  or  Well-Qualified) 
but  not  the  actual  score  (U.  S.  Department  of  Transportation 
Office  of  the  Inspector  General,  2010).  Currently,  the  CSPs  are 
no  longer  used  to  make  initial  selection  decisions,  option  track 
assignments  are  randomized,  and  specific  facility  placement 
decisions  are  delayed  until  completion  of  Academy  training. 

To  use  AT-SAT  for  placement  decisions,  AIR®  suggested 
computing  a  score  for  each  option,  based  on  the  option-specific 
regression  equations.  As  no  AT-SAT  validation  study  has  been 
conducted  specifically  for  TRACON  positions,  the  equation 
derived  from  the  tower  sample  was  used  to  represent  the  entire 
terminal  option.  The  applicants  would  then  be  assigned  to 
a  score  band  within  each  option.  For  example,  an  applicant 
could  be  classified  as  Well-Qualified  Terminal  and  Qualified  En 
Route  (or  vice  versa),  Well-Qualified  in  both,  or  Qualified  in 
both.  Current  use  of  AT-SAT  defines  Well-Qualified  as  a  score 
of  85-100,  Qualified  as  a  score  of  70-84.9,  and  Not  Qualified 
as  69.9  and  below. 

The  placement  procedure  suggested  by  AIR®  is  feasible  but 
has  three  drawbacks.  First,  the  overall  ranking  of  an  individual 
(which  impacts  hiring  decisions)  is  confounded  with  their 
ranking  within  an  option  (which  impacts  placement  deci¬ 
sions).  This  might  make  the  initial  selection  of  a  candidate 
more  complicated  and  less  systematic  with  more  judgment 
and  consideration  being  required  of  decision  makers  for  each 
individual  case.  For  example,  a  candidate  could  be  Qualified  in 
one  option  and  Well-Qualified  in  another.  If  all  positions  are 
filled  in  the  option  for  which  they  are  Well-Qualified,  will  they 
be  given  less  consideration  for  employment,  even  though  they 
might  have  been  categorized  as  Well-Qualified  overall  under 
previous  hiring  procedures?  In  the  end,  should  the  decision 
makers  consider  this  candidate  Qualified  or  Well-Qualified?  An 
even  more  complicated  situation  would  arise  if,  for  example,  an 
applicant  was  Well-Qualified  or  Qualified  in  one  option  and 
Not  Qualified  in  another. 
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Table  1 

Correlations  Between  Current,  En  Route,  and  Terminal  AT-SAT  Scores  and  1st  Facility  Success 


Current 

En  Route 

Terminal 

Current 

En  Route 

.880 

Terminal 

.651 

.793 

1st  Facility  Success 

.120 

.210 

.176 

Note.  All  correlations  significant  at p  <  .01,  n  =  2,332.  Current,  En  Route,  and  Terminal  AT-SAT  scores  are  based 
on  similar,  but  slightly  different  equations  developed  through  two  AT-SAT  validation  studies. 


Second,  given  the  width  of  rhe  categorical  bands  and  rhe 
correlation  found  between  the  current  en  route  score  (used  by 
AIR®  as  the  basis  for  en  route  placement)  and  the  tower  score 
(r  =  .65,  see  Table  1),  it  would  be  expected  that  if  the  placement 
rules  suggested  by  AIR®  were  used,  a  good  portion  of  candidates 
would  receive  the  same  categorical  ranking  for  both  options  (i.e., 
Well-Qualified  or  Qualified  in  both  options).  This  would  not 
provide  useful  information  on  which  to  base  a  placement  deci¬ 
sion,  and  the  decision  makers  would  be  required  to  judge  each 
case  individually,  considering  geographic  preferences  or  relying 
on  a  randomized  assignment  system. 

Third,  the  en  route  equation  was  reweighted  to  find  an 
optimal  balance  between  validity  and  the  reduction  of  ad¬ 
verse  impact,  but  the  tower  equation  reported  by  AIR®  was 
not  weighted  in  a  similar  way.  This  suggests  that  the  en  route 
equation  used  by  AIR®  in  their  analysis  would  produce  a 
different  option  score  for  reasons  other  than  “true”  subtest 
relationships  to  performance.  Part  of  the  difference  in  option 


scores  would  be  due  to  the  unique  weights  composing  each 
option  equation,  but  another  part  of  the  difference  in  option 
scores  could  be  attributed  to  the  adjustments  made  to  reduce 
adverse  impact  that  are  present  in  the  en  route  equation  and 
not  in  the  tower  equation. 

Taking  these  drawbacks  into  account,  we  investigated  an 
alternative  approach  to  placement.  The  first  step  would  be  to 
categorize  individuals  using  the  current  operational  AT-SAT 
equation,  which  was  weighted  to  mitigate  adverse  impact 
(Wise,  Tsacoumis,  Waugh,  Putka,  &  fforn,  2001),  into  Well- 
Qualified,  Qualified,  and  Not  Qualified  categories  using  the 
current  cut  scores  as  a  basis  for  initial  selection.  Second,  two 
additional  composite  scores  would  be  computed  based  on 
(a)  the  original,  unadjusted  weights  for  en  route  (Ramos  et 
al.,  2001a,  b),  and  (b)  the  tower  equation  developed  by  AIR® 
(2012)  for  terminal.  For  convenience,  these  will  be  referred 
to  as  the  Current,  En  Route,  and  Terminal  scores,  respectively, 
throughout  the  rest  of  this  paper. 
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Take  AT-SAT 


Weights  adjusted  to 

Compute 

mitigate  adverse  impact 

- ► 

Current 

(Wise  et  al.,  2001) 

Score 

Original  unadjusted 
En  Route  weights 
(Ramos  et  al.,  2001) 


If  hired,  place  into  option 
with  absolute  highest  score 


Applicant 

En  Route 

Terminal 

Placement 

1 

82 

79 

Q-ER 

2 

78 

84 

Q-T 

3 

72 

80 

Q-T 

Figure  1.  Hypothetical  Placement  Flowchart 
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New  Tower  weights 
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The  applicants  hiring  status  would  first  be  determined  by  using 
the  Current  score  to  determine  the  initial  categorical  rankings. 
Persons  categorized  as  “Not  Qualified”  on  the  basis  of  their  Cur¬ 
rent  score  would  be  removed  from  further  consideration.  Next, 
the  En  Route  and  Terminal  scores  would  be  computed  for  each 
person,  using  the  respective  option-specific  weights.  Whichever 
score  was  higher  would  serve  as  the  placement  recommendation, 
as  shown  in  Figure  1.  In  the  rare  event  of  a  tie,  the  applicant 
would  be  given  a  recommendation  of  “Either?  The  initial  cat¬ 
egorization  based  on  the  Current  score  would  then  be  attached 
to  this  option  recommendation. 

EVALUATION  OF  PROPOSED 
PLACEMENT  APPROACH 

The  following  analyses  were  conducted  to  evaluate  the  pro¬ 
posed  placement  approach.  First,  logistic  regression  analyses 
were  completed  to  verify  the  relationship  of  AT-SAT  scores 
(computed  using  the  three  equations)  to  first  facility  training 
success,  a  criterion  measure  not  used  in  the  two  previous  con¬ 
current,  criterion-related  validation  studies.  First  facility  train¬ 
ing  success  refers  to  whether  or  not  developmental  controllers 


achieved  certified  professional  controller  (CPC)  status  at  their 
first  facility.  Second,  cross-tabulations  were  computed  to  ex¬ 
amine  the  potential  outcomes  and  utility  of  using  AT-SAT  for 
placement.  Third,  given  that  placement  would  constitute  an 
employment  decision  encompassed  by  the  Uniform  Guidelines 
on  Employee  Selection  Procedures,  the  potential  for  adverse  impact 
was  assessed  against  the  4/5ths  rule  (29  C.F.R.  §  1607. 4D) 
(EEOC,  1978). 

The  data  used  for  these  analyses  were  extracted  from  FAA 
AVIATOR,  the  Air  Traffic  National  Training  Database  (NTD), 
the  AT-SAT  database,  and  the  FAA  Personnel  and  Payroll 
System  (FPPS).  Extracted  information  included  AT-SAT  test 
scores,  race,  gender,  pay,  and  developmental  training  status. 
The  sample  used  for  the  adverse  impact  analyses  included 
anyone  who  had  taken  AT-SAT  (N  =  18,663)  and  who  had 
race/gender  information  available  (Race:  N=  15,052;  Gender: 
N=  14,1 15).  The  sample  used  for  all  other  analyses  included 
individuals  who  had  AT-SAT  data  and  a  finalized  first  facility 
outcome  (i.e.,  achieved  CPC,  failed,  or  transferred  from  first 
facility  due  to  performance)  byjuly  2012  (N =  2,332).  In  both 
samples,  individuals  had  submitted  an  application  for  an  ATCS 
position  between  2007  and  2009. 
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Table  2 

Actual  Versus  Hypothetical  Placement 


Hypothetical  Placement 

Actual  Placement 

En  Route 

Terminal 

En  Route 

547 

(Correct  placement) 

297 

(Incorrect  placement) 

Terminal 

881 

(Incorrect  placement) 

607 

(Correct  placement) 

Table  3 

Cross-tabulation  of  Training  Completion  Rates  at  1st  Field  Facility 


Actual 

Placement 

Hypothetical 

Unsuccessful 

Successful 

Total 

Placement 

N 

% 

N 

% 

En  Route** 

En  Route 

111 

20% 

436 

80% 

547 

En  Route 

Terminal 

79 

27% 

218 

73% 

297 

Terminal 

En  Route 

153 

17% 

728 

83% 

881 

Terminal** 

Terminal 

159 

26% 

448 

74% 

607 

Total 

502 

22% 

1,830 

78% 

2,332 

Note.  **Indicates  “correct”  placement  -  meaning  that  applicants  were  actually  placed  in  the  option  that  AT-SAT 
would  have  predicted  had  it  been  used  for  this  purpose  at  the  time  of  hire. 


RESULTS 

Logistic  Regression 

The  results  of  the  logistic  regression  analyses  showed  that  the 
Current  -R2  =  .022,  y2  (1,  2332)  =  32.71  ,p  <  .001,  En  Route  - 
R2  =  .064,  y2  (1,  2332)  =  99.32,  p  <  .001,  and  Terminal—  R2  = 
.047,  y2  (1,  2332)  =  71.88 ,p  <  .001  scores  (based  on  the  previ¬ 
ously  derived  equations)  were  statistically  significant  predictors 
of  first  facility  training  success.  The  raw  correlations  between 
these  scores  and  first  facility  training  success,  uncorrected  for 
range  restriction,  were  similar  but  not  identical  to  each  other 
and  can  be  found  in  Table  1.  These  findings  parallel  the  results 
obtained  during  both  concurrent  validation  studies  to  assess 
the  predictive  validity  of  AT-SAT  using  ordinary  least  squares 
regression  analyses  and  other  types  ofjob  performance  measures 
(i.e.,  scores  on  the  BSS  and  TSBPM),  as  well  as  the  results  of  a 
longitudinal  validation  of  AT-SAT  using  first  facility  training 
success  as  the  criterion  (see  Broach  et  al.,  2013).  Additionally, 
when  logistic  regression  analyses  were  run  separately  for  the 
En  Route  and  Terminal  samples  (not  restricting  subtest  weights 
based  on  the  previously  derived  equations),  the  subtest  scores 
were  differentially  correlated  with  first  facility  training  success 
similar  to  the  findings  of  AIR®  that  the  subtest  weights  using  a 
sample  of  tower  controllers  were  not  identical  to  those  found 
in  the  original  en  route  validation  study.  Taken  together,  this 
evidence  demonstrates  some  degree  of  differential  validity  (i.e., 
overall  validity  coefficients  are  different  by  option  samples)  and 
prediction  (i.e.,  the  weights  of  each  subtest  vary  for  each  op¬ 
tion  sample),  both  technical  requirements  as  described  by  the 
Standards  for  Educational  and  Psychological  Testing  (American 


Educational  Research  Association,  American  Psychological 
Association,  &  National  Council  on  Measurement  in  Educa¬ 
tion,  1999),  as  well  as  the  Principles  for  the  Validation  and  Use 
of  Employee  Selection  Procedures  (SIOP,  2003).  In  other  words, 
based  on  the  considerations  encompassed  by  these  standards 
and  principles,  AT-SAT  meets  the  technical  requirements  and 
can  be  further  considered  for  use  in  the  placement  of  newly 
hired  controllers  into  options.  These,  however,  are  not  the  only 
considerations  that  need  to  be  weighed  by  the  FAA  before  using 
AT-SAT  for  placement. 

Cross-Tabulation  Analysis 

The  next  step  in  the  analysis  was  to  cross-tabulate  actual  vs. 
hypothetical  placement.  “Actual  Placement”  was  the  official 
assignment  of  newly  hired  controllers  to  en  route  or  terminal 
facilities  without  regard  to  their  Current  score  on  AT-SAT.  “hypo¬ 
thetical  Placement”  was  the  decision  that  would  have  been  made 
using  the  En  Route  and  Terminal  scores  derived  from  AT-SAT. 
There  were  four  possible  combinations  of  actual-by- hypothetical 
placements  (Table  2).  The  cross-tabulation  compared  those 
who  were  hypothetically  placed  “correctly”  or  “incorrectly”  in 
terms  of  their  success  in  field  training  (Table  3).  Placement  was 
“correct”  when  the  actual  placement  matched  the  hypothetical 
placement  based  on  En  Route  and  Terminal  scores;  placement 
was  “incorrect”  when  the  actual  placement  did  not  match  the 
hypothetical  placement.  For  example,  297  cases  were  incorrectly 
placed  in  the  en  route  option  that  should  have  been  placed  into 
the  terminal  option  based  on  their  En  Route  and  Terminal  scores. 
Similarly,  881  incorrect  actual  placements  were  made  into  the 
terminal  option  (Table  2). 
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Table  4 

Training  Success  Rates  at  the  1st  Field  Facility  With  and  Without  Placement 


Success  Rate 

Success  Rate  With 

Without  Placement 

Placement 

En  Route  (36%  of  positions) 

77% 

80% 

Terminal  (64%  of  positions) 

79% 

77%* 

Across  Options  (weighted  by  number  of  positions) 

78.28% 

78.08% 

Note.  ‘Indicates  rate  adjusted  for  likelihood  of  filling  40%  of  terminal  positions  with  applicants  initially 
recommended  for  En  Route  placement 


A  second  cross-tabulation  was  computed  for  those  who 
were  placed  “correctly”  or  ‘incorrectly”  by  their  success  in  field 
training  (Table  3),  and  the  results  were  adjusted  based  on  the 
typical  proportion  of  hires  assigned  to  each  option  (Table  4). 
The  results  of  this  analysis  suggest  that  the  utility  of  using  AT- 
SAT  for  placement  is  marginal  to  slightly  negative,  depending 
on  how  the  data  are  examined.  The  FAA  could  potentially  see  a 
3%  increase  to  80%  in  the  success  rate  of  controllers  “correctly” 
placed  into  the  en  route  option,  as  compared  to  the  baseline 
success  rate  (without  AT-SAT  guided  placement)  of  77%.  How¬ 
ever,  this  gain  could  be  offset  by  a  5%  reduction  to  74%  in  the 
success  rate  of  those  “correctly”  placed  into  the  terminal  option, 
as  compared  to  a  baseline  success  rate  (without  AT-SAT  guided 
placement)  of  79%  (see  Table  3),  for  a  net  reduction  in  success 
rates  across  both  options  of  2%.  However,  this  loss  must  be 
reexamined  and  weighted  within  the  context  of  the  number  of 
positions  available  in  each  option  and  the  number  of  controllers 
being  hypothetically  placed  in  the  en  route  option.  The  overall 
baseline  success  rate  in  terminal  without  placement  is  driven 
upwards  by  the  higher  success  rate  of  individuals  that  would 
hypothetically  have  been  placed  in  the  en  route  option.  Given 
the  number  of  applicants  that  scored  higher  on  the  En  Route 
equation,  as  compared  with  the  number  of  positions  typically 
available  for  en  route  controllers  in  recent  years,  it  is  estimated 
that  approximately  40%  of  available  terminal  positions  could 
be  filled  by  individuals  with  en  route  recommendations.  This 
would  likely  be  the  preferred  policy,  given  their  apparent  ability 
to  succeed  in  either  option. 

Thus,  to  accurately  estimate  the  overall  success  rate  with  AT- 
SAT  guided  placement  for  terminal,  given  the  likely  situation 
that  40%  of  the  positions  could  be  filled  by  applicants  scoring 
higher  on  the  En  Route  equation  (who  would  likely  have  higher 
success  rate  -  83%  vs.  74%),  a  weighted  average  was  computed. 
The  overall  success  rate,  assuming  placement  of  some  applicants 
with  En  Route  placement  recommendations  into  the  terminal 
option  then  becomes  77%  [(83%  success  rate  x  40%  of  the 
positions)  +  (74%  success  rate  x  60%  of  the  positions)]  instead 


of  74%  for  terminal  positions.  This  computation  results  in  a 
success  rate  2%  lower  than  the  current  terminal  success  rate 
seen  without  using  AT-SAT  for  placement. 

In  sum,  if  AT-SAT  is  used  to  guide  placement  by  option,  there 
is  a  potential  increase  in  success  rates  for  those  placed  in  en  route 
of  3%  but  a  potential  decrease  for  those  placed  in  terminal  of 
2%,  for  an  overall  1%  increase  in  success  rates.  However,  this 
estimate  must  also  be  considered  within  the  context  of  the  ratio 
of  people  hired  into  each  option.  Generally  speaking,  because 
more  people  are  hired  into  the  terminal  option  (accounting  for 
approximately  64%  of  open  positions  yearly),  the  decrease  in 
the  terminal  success  rate  must  be  weighed  more  heavily  in  the 
calculation  of  overall  success  rates  computed  with  and  without 
placement  (Table  4).  Taking  the  higher  hiring  rate  in  the  termi¬ 
nal  option,  the  net  effect  of  using  AT-SAT  for  placement  would 
likely  be  a  very  slight  reduction  in  the  overall  success  rate  across 
both  options  (Table  4). 

Adverse  Impact  Analysis 

As  with  other  employment  decisions,  a  placement  decision 
carries  with  it  the  potential  to  impact  an  individual’s  ability 
to  earn.  Given  the  nature  of  this  decision,  the  potential  for 
adverse  impact  against  members  of  protected  groups  must  be 
considered.  Using  data  from  FPPS,  it  was  determined  that  en 
route  controllers  earn  on  average  approximately  $20K  more  per 
year  than  terminal  controllers.  The  difference  in  annual  salaries 
was  calculated  using  a  snapshot  of  the  FPPS  data  captured  in 
July  2012.  This  computation  produced  a  very  rough  estimate 
calculated  across  all  levels  of  facilities  and  is  not  intended  to 
estimate  actual  losses  or  gains  an  individual  controller  might 
experience.  Many  other  factors  that  could  not  be  measured 
here  would  help  determine  actual  losses  or  gains  for  each  in¬ 
dividual.  Regardless,  on  average ,  receiving  a  recommendation 
for  placement  into  the  en  route  option  would  likely  provide 
an  individual  with  a  greater  opportunity  to  earn  more  over  the 
course  of  employment  and  is  thus  considered  the  preferred 
option  for  calculating  adverse  impact. 
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Table  5 

Adverse  Impact  From  Placement  Decision 


Hypothetical  Placement 

En  Route 
Placement 
Rate 

Adverse 

Impact 

Ratio3 

En  Route 

Terminal 

Total 

By  Ethnicity 

Asian 

228 

228 

456 

.50 

.95 

Black 

713 

2,324 

3,037 

.23 

.45 

Hawaiian-Pacific  Island0 

26 

49 

75 

.35 

.66 

Hispanic-Latino 

269 

556 

825 

.33 

.62 

Native  American-Alaskan  Native 

30 

35 

65 

.46 

.88 

White 

4,632 

4,209 

8,841 

.52 

Multi-racial 

462 

569 

1,031 

.45 

.86 

No  groups  marked 

357 

358 

715 

.50 

.95 

Total 

6,717 

8,328 

15,045 

By  Sex 

Female 

1,103 

2,320 

3,423 

.32 

.66 

Male 

5,350 

5,686 

11,036 

.48 

Total 

6,453 

8,006 

14,459 

Note.  “Adverse  impact  ratio  calculated  with  respect  to  whites  for  ethnicity  and  male  for  sex. 

bGroups  comprising  less  than  2%  of  the  applicant  pool  are  italicized.  Bold  ratios  are  less  than  what  acceptable  under 
the  4/5 ths  Rule  (0.80). 


Using  the  placement  rules  previously  described,  assigning 
controllers  to  an  option  using  their  AT-SAT  scores  could  result 
in  differential  placement  rates  by  race  and  sex  into  the  terminal 
and  en  route  options  (Table  5).  For  example,  just  23%  of  black 
candidates  would  be  recommended  for  placement  in  en  route, 
compared  to  52%  of  white  candidates  (adverse  impact  ratio  = 
.23/. 52,  or  .45,  where  the  threshold  for  adverse  impact  is  defined 
as  a  ratio  of  .80  or  less  by  the  Uniform  Guidelines  on  Employee 
Selection  Procedures  [EEOC,  1978]).  The  adverse  impact  ratio  for 
Hispanic/Latino  applicants  was  .62  and  for  females  was  .66.  This 
adverse  impact  would  result  in  addition  to  the  adverse  impact 
these  protected  groups  already  face  in  assignment  to  the  Well- 
Qualified  category  ranking  for  initial  selection  considerations. 

SUMMARY 

Looking  at  both  of  the  concurrent  validation  studies  and  this 
current  set  of  analyses  together,  there  is  sufficient  evidence  to 
suggest  that  the  abilities  required  to  perform  air  traffic  control 
tasks  do  vary,  to  some  limited  degree,  by  option.  The  regression 
analyses  (calculated  repeatedly  using  different  samples  and  at 
different  times)  have,  in  fact,  derived  different  equations  for  the 
two  options,  which  overlap  but  are  not  completely  identical.  This 
evidence  can  help  provide  the  technical  justification  required, 
if  the  FAA  were  to  pursue  the  use  of  AT-SAT  for  placement. 

However,  it  is  not  clear  that  the  variation  by  option  is  of  a  suf¬ 
ficient  degree  to  j  ustify  differential  placement,  given  the  minimal 
utility  observed.  Moreover,  the  utility  of  using  AT-SAT  to  guide 
placement  is  minimal  -  and  might  be  slightly  counterproductive 


for  the  FAA.  The  cross-tabulations  indicated  that  the  success  rate 
in  en  route  would  increase  if  AT-SAT  is  used  for  placement  but 
would  decrease  in  terminal.  Taken  across  both  options,  field 
training  success  rates  would  not  likely  change  in  a  meaningful 
way,  provided  that  the  number  of  candidates  typically  hired  for 
each  option  in  recent  years  remains  consistent. 

Additionally,  in  both  this  study  and  the  AIR®  (20 1 2)  analysis, 
the  AT-SAT  equation  derived  from  a  sample  of  tower  controllers 
was  used  to  represent  all  of  the  terminal  option,  because  no  data 
were  available  examining  TRACON-only  controllers.  Given  a 
similar  reliance  on  radar  technology,  as  well  as  increased  job 
complexity,  ATCSs  working  in  stand-alone  TRACONs  might 
be  more  similar  to  en  route  controllers  than  to  tower  control¬ 
lers.  In  other  words,  AT-SAT  subtest  score  relationships  with 
performance  for  TRACON  controllers  may  be  more  similar  to 
those  for  en  route  than  to  tower  controllers.  If  the  use  of  AT-SAT 
for  placement  purposes  were  to  be  further  pursued  by  the  FAA, 
it  is  recommended  that  data  be  collected  on  TRACON-only 
controllers  to  determine  first,  if  AT-SAT  is  also  a  valid  predic¬ 
tor  of  TRACON  performance  in  general  —  a  highly  likely  result 
given  the  outcomes  of  previous  validation  studies  on  AT-SAT, 
as  well  as  job  analyses  suggesting  similar  worker  requirements. 
And  second,  if  TRACON-only  controllers  can,  in  fact,  be  placed 
using  the  equation  derived  from  tower  controllers,  or  if  the  en 
route  equation  would  be  more  suitable,  given  the  similar  nature 
of  the  work.  It  might  be  found  that  they  are  more  accurately 
represented  using  the  original  en  route  equation,  or  that  there 
are  substantial  differences  between  all  three  jobs,  andTRACON 
controllers  require  an  entirely  separate  equation. 


7 


Finally,  placement  using  AT-SAT  could  potentially  have  adverse 
impact  on  individuals  in  protected  classes.  That  is,  members  of  pro¬ 
tected  classes  would  be  placed  into  higher  paying  en  route  facilities 
at  less  than  80%  of  the  rate  of  majority  members  of  each  class  (race 
and  gender) .  Differential  placement  rates  on  the  basis  of  AT-SAT 
scores  could  create  troubling  pay  disparities  by  race  and  sex.  If 
the  FAA  were  to  use  AT-SAT  for  placement,  the  risk  of  additional 
adverse  impact  and  pay  disparities  should  be  evaluated  against  the 
marginal  utility  of  placement  in  terms  of  changes  in  field  training 
success  rates.  In  sum,  given  the  findings  of  both  validation  studies, 
the  analyses  conducted  here,  and  the  projected  ratio  of  controllers 
that  will  likely  be  hired  into  each  option,  using  AT-SAT  scores  to 
guide  placement  decisions  is  not  recommended  at  this  time. 
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